Papers by Jann Railey Montalan
SEA-BED: How Do Embedding Models Represent Southeast Asian Languages? (2026.acl-long)
Copied to clipboard
Wuttikorn Ponwitayarat, Peerat Limkonchotiwat, Raymond Ng, Jann Railey Montalan, Thura Aung, Jian Gang Ngui, Yosephine Susanto, William Chandra Tjhi, Panuthep Tasawong, Erik Cambria, Ekapol Chuangsuwanich, Sarana Nutanong
| Challenge: | SEA-BED examines how multilingual text embeddings perform across tasks and languages . performance gaps arise from data coverage, training objectives, and architectural design, authors say . |
| Approach: | They propose a large-scale benchmark covering 10 SEA languages and diverse embedding tasks. |
| Outcome: | The proposed model performs poorly across languages and tasks, but language-task analyses reveal inconsistencies . the results suggest that performance gaps arise from limitations in data coverage, training objectives, and architectural design. |
Batayan: A Filipino NLP benchmark for evaluating Large Language Models (2025.acl-long)
Copied to clipboard
Jann Railey Montalan, Jimson Paulo Layacan, David Demitri Africa, Richell Isaiah S. Flores, Michael T. Lopez Ii, Theresa Denise Magsajo, Anjanette Cayabyab, William Chandra Tjhi
| Challenge: | Recent advances in large language models (LLMs) have demonstrated remarkable capabilities on widely benchmarked high-resource languages. |
| Approach: | They propose a benchmark that systematically evaluates LLMs across three key natural language processing competencies: understanding, reasoning, and generation. |
| Outcome: | The proposed benchmark covers eight tasks covering Tagalog and code-switched Taglish utterances. |
SEA-HELM: Southeast Asian Holistic Evaluation of Language Models (2025.findings-acl)
Copied to clipboard
Yosephine Susanto, Adithya Venkatadri Hulagadri, Jann Railey Montalan, Jian Gang Ngui, Xianbin Yong, Wei Qi Leong, Hamsawardhini Rengarajan, Peerat Limkonchotiwat, Yifan Mai, William Chandra Tjhi
| Challenge: | Existing LLM benchmarks are capable of evaluating specific capabilities in English as well as in various mid- to low-resource languages, but a comprehensive and culturally representative evaluation suite for the SEA languages has not been developed thus far. |
| Approach: | They propose a holistic linguistic and cultural LLM evaluation suite that emphasizes SEA languages and introduces a leaderboard that allows users to understand models’ multilingual and multicultural performance. |
| Outcome: | The proposed evaluation suite emphasizes SEA languages and supports Filipino, Indonesian, Tamil, Thai, and Vietnamese. |